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November  5,  1979  to  December  31,  1979.  The  Tasks/Objectives 
and/or  Purposes  of  the  overall  project  are  connected  with 
the  design,  development,  demonstration  and  transfer  of 
advanced  command  and  control  (C2)  computer-based  systems; 
this  report  covers  work  in  the  computer-based  development 
and  demonstration  areas  only.  The  Technical  Problems  thus 
addressed  include  providing  support’Tor"^5'2“systemS"lIevelop- 
ment  in  the  information,  decision,  forecasting,  training  and 
readiness  areas  via  computer  timesharing  and  demonstrations 
and  how  to  accelerate  development  through  structured  documen¬ 
tation.  The  General  Methods  employed  have  involved  building 
and  maintaining  a  dual  PDP  Tl/70  timesharing  system  and 
simultaneously  configuring  multiple  microcomputer  systems. 

In  addition,  problems  have  been  approached  via  the  structuring 
of  a  realistic  demonstration  and  test  environment.  Technical 
Results  to  date  include  a  seven  day,  twenty  four  hour  a  day 
C2 development  support  system,  the  development  of  realistic 
testing  and  demonstration  scenarios  and  capabilities,  and 
the  completion  of  a  plan  for  the  documentation  of  C2  system 
development.  The  Hardware  Configuration  utilized  for  this 
research  is  entirely  GFE.  The  systems  software  includes  the 
UNIX  operating  system,  FORTRAN  IV  PLUS,  C,  and  some  limited 
statistical  packages;  the  applications  software  consists  of 
numerous  programs  in  the  C2  information,  decision,  forecasting, 
training  and  readiness  areas.  Future  Research  will  be  con¬ 
ducted  in  the  C2  computer-based  systems  design  and  transfer 
areas . 
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1.0  INTRODUCTION 


1.1  Problem  Statement 

The  Defense  Advanced  Research  Projects  Agency  s 
Cybernetics  Technology  Office  (DARPA/CTO)  has  as  its  pri¬ 
mary  mission  the  development,  application,  and  transfer  of 
computer-based  systems  for  improved  Department  of  Defense 
(DoD)  information  mangement  and  display,  forecasting,  de¬ 
cision  making,  training  and  human  performance,  especially 
as  such  activities  occur  in  Command  and  Control  (C2)  envi¬ 
ronments.  Unfortunately,  however,  the  research  and  develop¬ 
ment  (R&D)  process  connected  with  the  development  of  advanced 
C2  computer-based  systems  is  fraught  with  problems.  Speci¬ 
fically,  such  problems  may  be  categorised  as  follows: 

1.1.1  Computer-based  Systems  Design 

1.1. 1.1  Neglect  of  "front-end"  analysis.  This  prob¬ 
lem  runs  rampant  throughout  all  of  the  projects  that  have  as 
a  final  product  either  computer-based  systems  software  or 
analytic  or  descriptive  data  sets.  Moreover,  front-end 
analysis  has  seldom  been  conducted  in  any  of  the  areas  of 
the  computational  needs  of  the  hardware  and  software  spectrum. 
Neglect  of  such  analysis  invites  disaster  and  circumvents 
normally  acceptable  programming  practices.  An  example 
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illustrating  the  myriad  of  problems  that  can  occur  with¬ 
out  proper  design  analysis  is  the  current  state  of  the 
Terrorism  Research  and  Analysis  Project  (TRAP).*  TRAP 
software  is  a  TEKTRONIX  4051  resident  BASIC  program. 

Many  demonstrations  of  its  capabilities  show  the  extreme 
value  it  has  as  a  sophisticated  Indications  and  Warnings 
(I&W)  and  operations  research  system.  However,  it  is 
designed  to  sun  on  a  very  specific  type  of  graphics  micro¬ 
processor  for  which  there  is  only  one  manufacturer.  It 
can  also  only  be  demonstrated  on  large  screen  projection 
via  a  Hughes  scan  converter  and  a  specially  modified  4051 
(of  which  there  are  only  two  in  the  Washington,  D.C.  area). 
"Front-end  analysis"  was  neglected  in  this  example.  Had 
it  been  conducted  such  potentially  costly  oversights  in 
hardware  and  software  design  might  have  been  prevented. 

(This  example  is  not  intended  to  undermine  the  research 
efforts  of  specific  individuals  or  organizations,  but 

rather  to  illustrate  a  critical  problem  in  the  design  of 

,  2 

complicated  computer-based  systems.) 

1.1. 1.2  Expensive  and  fundamental  "disconnects"  among 
the  programmer,  the  intended  product,  and  the  ultimate  user. 
So  often,  through  a  very  basic  misunderstanding  in  the  design 
phase,  there  occurs  a  strange  phenomenon  which  seperates  the 
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intended  purpose  of  the  reseach  tool  from  its  preparation 
and  construction.  This  "distance"  often  causes  enormous 
problems  in  the  final  stages  of  software  implementation 
and  transfer.  If  in  fact,  the  product  delivered  is 
neither  what  was  intended  nor  what  can  be  used,  it  must  be 
re-written,  re-tested,  re-validated  and  re-documented-- 
generally  a  very  expensive  process.  The  cost  in  man-hours 
alone  is  sometimes  staggering.  Further  costs  of  late- 
delivery,  and  other  projects  suffering  because  of  a  re¬ 
shuffling  of  priorities  are  also  not  inconsequential.  It 
is  important  to  keep  sight  of  who  the  ultimate  user  is, 
where  the  tool  will  be  utilized,  when  it  must  be  ready  to 
be  effective,  and  how  it  should  be  implemented.  For  example, 
if  the  product  is  a  low  cost,  short  lead-time  one,  it  need 
not  go  through  a  rigorous  design  stage  once  the  above 
criterion  are  met.  Restated,  if  the  product  is  to  be 
"quick  and  dirty"  this  fact  should  predict  to  overriding 
developmental  techniques.  However,  these  are  the  only 
kinds  of  products  which  should  be  allowed  to  slip  through 
an  intensive  design  critique.  Examples  demonstrating 
this  "fundamental  disconnect"  are  plentiful;  e.g.  the 
Early  Warning  and  Monitoring  System  (EWAMS)  was  produced 
with  great  care  and  planning.  The  goal  was  to  create  a 
unique  monitoring  and  forecasting  system  that  would  be 
both  focused  and  easy  to  use  by  the  intelligence  community, 
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especially  through  the  Defense  Intelligence  Agency's 
National  Military  Intelligence  Center  (DIA/NMIC). ^ 

The  (interim)  product  transferred  to  the  DIA/NMIC  had 
no  user  manual;  it  used  too  many  research  and  statistical 
terms;  it  did  not  reflect  the  needs  of  a  daily  "watch" 
analyst;  and  it  was  not  written  so  that  it  could  be  trans¬ 
ferred  easily  to  a  non-UNIX*,*  non- timesharing ,  and  heavily 
utilized  DIA/NMIC  computer  system.  Again,  the  purpose  is 
not  to  undermine  legitimate  efforts,  but  rather  to  point 
out  the  critical  necessity  of  exacting  procedures  that 
must  be  followed  early  in  the  design  phase,  and  that  a 
"fundamental  disconnect"  between  the  developer  and  user 
can  increase  transfer  cost  by  orders  of  magnitude. 

1.1. 1.3  Non-standardized  data  sets  and  codebooks. 

In  the  early  years  of  the  conceptualization  and  creation 
of  the  Demonstration  and  Development  Facility  (DDF) ,  it 
became  evident  that  a  large  portion  of  the  DDF  user  com¬ 
munity  would  be  tasked  with  the  creation  and  maintenance 
of  various  data  sets.  This  function  has  no  less  impor¬ 
tance  than  the  analytic  software  tools  which  often  evolve 
from  such  data  sets.  But,  here  too  we  find  design  flaws. 
Care  should  have  been  taken,  at  the  outset,  to  standardize 
the  coding  and  collection  of  the  data,  particularly  with  a 
view  to  how  they  might  later  be  analyzed  and  processed  via 


4 


a  computer-based  delivery  system.  Two  examples  of  where 
improper  data  standardization  in  the  design  phase  resulted 
in  unnecessary  man-hour  efforts  include  the  Cross  National 
Crises  Indicators  (CNCI)5  project,  which  collected  a  very 
specialized  data  set  consisting  of  "surprise  attack"  event 
data.  The  data  collected  was  intended  to  follow  World 
Event  Interactions  Survey  (WEIS)  codebook  techniques; 
however,  the  resultant  data,  differed  so  severely  from 
pure  WEIS  data  that  a  special  version  of  the  EWAMS  program 
had  to  be  adapted  to  utilize  desired  indicators  and  graphics. 
The  modification  of  EWAMS  to  fit  the  "surprise  attack"  data 
was  a  relatively  small  task,  but  costly  nonetheless.  In 
the  second  case,  data  driving  the  CACI ,  Inc.  U.  S.  Executive 
Aids  (EXECAID)  had  been  so  well  designed,  it  could  easily 
fit,  in  its  entirety,  on  one  TEKTRONIX  4051  casette  tape. 

But  problems  resulted  when  members  of  the  Inter-university 
Consortium  for  Political  Social  Research  (ICPSR)  wanted 
copies  of  the  data  only,  and  found  that  it  was  uniquely 
tied  to  the  software  with  which  it  is  used.  In  both  cases 
errors  in  data  set  standardization  occurred  causing  delays 
and  expensive  efforts  to  correct. 

1.1.2  Computer-based  Systems  Development 

1. 1.2.1  Hardware .  Here  selection  and  usage  problems 
are  often  encountered  in  many  of  the  more  advanced  research 


projects.  Research  products  that  use  ineffective  computer 
systems  will  fail  no  matter  how  excellent  the  value  of  the 
research  product.  Correct  hardware  selection  is  critical 
to  the  successful  development  and  transfer  of  a  research 
product.  Factors  such  as  portability ,  maintenance  costs, 
backup  systems,  commercial  availability,  and  life-expectancy 
are  all  important  to  the  hardware/software  marriage.  For 
example,  the  earliest  application  of  the  Perceptronics , 

Inc.,  Ultra-Rapid  Reader  (URR)  was  written  on  the  DDF  using 
a  TEKTRONIX  4025  graphics  terminal.®  The  problem  with  this 
selection  was  the  combination  of  phosphorous  persistance 
and  character  size  for  projection.  Even  though  this  was  a 
pilot  application  much  of  its  success  depended  upon  the 
readability  of  the  output.  The  method  of  "hardware  selec¬ 
tion  by  availability"  is  insufficient  when  the  hardware 
inhibits  the  applications  software.  It  also  is  extremely 
important  to  address  the  data  requirements  as  they  apply  to 
hardware  selection.  Problems  in  this  area  have  plagued  the 
DDF  since  its  inception.  For  example,  the  size  alone  of  the 
WEIS  data  set  is  so  large  that  in  many  instances  the  data 
set  required  so  much  DDF  disk  space  that  there  wasn't  suf¬ 
ficient  space  remaining  to  permit  further  software  develop¬ 
ment  on  either  EWAMS  or  other  projects.7  Proper  product 
design  and  development  could  have  by-passed  these  problems. 

1.1. 2. 2  Software.  Here,  development  problems  arise 
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throughout  all  programming  activities  in  every  organisation. 
Poor  software  development  procedures  result  in  much  wasted 
time,  effort  and  cost.  Typical  problems  concentrate  usually 
around  language  selection  and  implementation.  Many  langu¬ 
ages  are  incompatible  with  one  another.  This  problem  of 
imcompatibility  exists  not  only  between  operating  systems 
(such  as  UNIX  vs.  RSX11-M)  but  on  computer  systems  such  as 
HONEYWELL  Level  6  vs.  DEC  11/70  as  well.  For  example,  most 
UNIX  trained  systems  programmers  use  a  term  called  "vanilla 
UNIX".  Vanilla  UNIX  is  the  basis  for  the  next  level  of. in¬ 
compatible  versions  of  the  same  Bell  Laboratories  software. 
The  others,  "BBN  UNIX",  Rand  UNIX",  "NSA  UNIX"  (closest  to 
vanilla)  and  "ISC  UNIX"  all  differ  so  dramatically  that  it 
becomes  difficult  to  easily  transport  user  or  system  soft¬ 
ware  from  one  UNIX  system  to  another.®  There  are  of  course 
literally  hundreds  of  modifications  of  the  UNIX  version  6 
operating  system.  This  may  all  be  corrected  with  the 
release  of  version  7  UNIX  and  PWB  UNIX,  but  history  will 
most  likely  repeat  itself  and  lend  more  versions  of  more 
levels  of  the  same  complicated  operating  system.*®  This 
problem  (multi-UNIX's)  is  further  complicated  by  the  fact 
that  UNIX  has  the  ability  to  support  many  compilers, 
interpreters  and  assembly  languages.  In  fact  UNIX  has  a 
language  'C'  and  YACC — The  ' C  System  Compiler  and  "Yet 
Another  Compiler  Compiler"!  The  languages  of  the  system 
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could  include  three  or  four  BASIC's,  AS,  RATFOR,  FOR,  FC, 
F4P,  PASCAL,  LISP,  and  so  on.  The  problem  is  real,  and  a 
pragmatic  approach  must  be  taken  to  proper  software  language 
development.  Decisions  must  be  made  at  the  operating  sys¬ 
tems  level  as  to  the  availability  of  S/W  facilities  such 
as  DoDl.**  Decisions  must  be  made  when  and  how  DoDl  should 
be  addressed.  For  example,  what  research  now  under  develop¬ 
ment  will  still  be  so  when  DoDl  becomes  universal? 

Pragmatic  decisions  must  be  made  not  only  as  to  the 
user  availability  of  such  languages  but  the  fundamental 
choice  of  a  language  for  an  application.  For  example, 
if  a  short  term  project,  with  a  specific  transfer  applica¬ 
tion  requires  APL,  then  it  should  be  used.  But,  on  the 
other  hand,  if  the  need  for  the  resultant  research  tool  is 
more  universal,  then  it  would  be  wrong  to  allow  programming 
to  begin  in  APL.  For  example.  Decisions  and  Designs,  Inc. 
and  the  Computer  Corporation  of  America,  Inc.  constructed 

a  very  valuable  set  of  decision  making  and  evaluation  aid- 

12 

ing  tools  for  the  U.  S.  Marine  Corps.  The  results  were 
spectacular  but  the  software  was  written  in  APL,  and  the 
Marines  could  only  accept  transfer  of  software  written  in 
COBOL. 


Also  under  the  jurisdiction  of  software  development 
comes  the  very  important  topic  of  documentation.  Throughout 
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the  development  phase  of  either  software  tools  or  data  sets, 
time  must  be  spent  on  building  effective  documentation. 
Documentation  takes  the  forms  of  internal  documentation, 
systems  specifications,  users  manuals,  and  codebooks. 
Improperly  documented  code,  or  code  without  documentation 
lives  only  as  long  as  its  author.  It  loses  its  efficiency; 
it  can  no  longer  be  maintained;  it  becomes  forgotten.  The 
cost  of  rewriting  software,  because  the  documentation  no 
longer  exists,  is  prohibitive.  For  example,  the  first 
release  of  the  BBN  "steamer"  program  to  the  DDF  had  no  users 
manual  and  no  system  specifications.  The  program  could  only 
be  run  by  guess  work,  and  modifications  to  it  were  diffi¬ 
cult  and  time-consuming. 

1.1. 2. 3  Coordination.  Far  too  often  redundant  code, 
or  redundancy  of  effort  exists  in  program  development.  The 
problem  comes  about  partially  through  the  "not  invented 
here"  syndrome.  Competitive  researchers  do  not  wish  to 
admit  that  there  may  be  others  that  have  attacked  a  similar 
problem  with  success.  Therefore,  they  close  their  minds 
to  the  existence  of  a  solution  to  the  problem  at  hand.  In 
other  cases  the  "re-invent  the  wheel"  syndrome  applies.  By 
sheer  fact  that  competition  exists  between  contractors, 
inventions,  new  ideas  and  methods  are  not  shared  among  the 
user  community.  Result:  the  wheel  is  re-invented  over  and 


9 


over  end  over  again.  An  example  of  lack  of  coordination 
can  be  seen  in  data  collection  efforts  by  both  the 
Brookings  Institution  and  CACI,  Inc.1^  Both  research 
projects  collected  similar  crisis  data  sets.  At  no  time 
during  these  projects  was  there  any  interfacing  of  data 
or  ideas.  At  the  end  of  these  projects  the  DDF  received 
two  tapes,  one  from  each  contractor,  both  similar,  and 
neither  acknowledging  the  existence  of  the  other.  Other 
examples  of  poor  coordination  exist.  Software  written  at 
the  University  of  Southern  California  (USC)  to  allow  ef¬ 
ficient  terminal  input  for  data  collection  and  management 

14 

was  installed  on  the  DDF  for  use  with  the  WEIS  data  set. 
However,  many  of  these  utilities  are  not  used,  causing 
wasted  efforts,  on  the  part  of  others  who  could  use  the 
techniques  for  WEIS  and  other  data  collection. 

Falling  also  under  the  heading  of  poor  dissemination 
and  coordination,  is  the  management  of  research  tools  for 
internal  development.  Part  of  the  DARPA/CTO  mission  is  the 
development  of  computer-based  systems  for  improved  informa¬ 
tion  management,  display,  forecasting,  decision  making, 
training  and  human  performance  research  products.  These 
new  tools  could  be  used  internally  by  the  researchers  to 
help  achieve  a  "bubbling-up"  or  "percolator"  effect.  For 
example,  Decisions  and  Designs,  Inc.  constructed  many  useful 
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tools  in  decision  making.  Many  of  these  tools  such  as 
the  Multi-Attribute  Utility  (MAU)  program  could  be  used 
to  make  more  efficient  research  decisions. 

Another  problem  in  coordination  is  the  "closed  com¬ 
munity"  syndrome.  Too  often  researchers  within  the  CTO 
program  refuse  to  look  beyond  work  outside  of  their  own 
community.  For  example,  Artifical  Intelligence  (AI) 
techniques  developed  by  the  Information  Processing 
Techniques  Office  (IPTO)  may  be  useful  to  CTO  researchers. 
It  is  cost  effective  to  glean  all  relevant  work  regardless 
of  the  sponsoring  office,  agency  or  department. 

1.1.3  Demonstrations 


1.1. 3.1  Opposing  "cultures".  Contractors  typically 
by  the  nature  of  their  roles  as  researchers  are  disconnected 
from  the  the  nature  of  organizational  decision  making.  As 
a  result,  contractors  are  seldom  able  to  provide  the  neces¬ 
sary  context  for  an  effective  demonstation.  This  is  not 
meant  that  they  cannot  (or  are  unwilling  to)  bridge  the  gap 
between  product  development  and  application.  Instead,  it 
is  to  highlight  the  issue  of  "comparative  advantages" . 
Researchers  are  good  at  what  they  do,  but  seldom  can  make 
an  effective  leap  from  development  to  application.  They  are 
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often  at  a  comparative  disadvantage  when  they  try.  Without 
this  leap  however,  work  may  be  rendered  useless  and  inef¬ 
fective.  Here,  the  real  value  of  a  Demonstration  and 
Development  Facility  (DDF) — cognizant  of  the  critical  dis¬ 
tance  between  the  researcher  and  the  "user" — comes  into 
play.  Since  contractors  are  generally  unskilled  in  the 
art  of  interacting  effectively  with  government,  military 
and  civilian  personnel  at  all  levels,  they  harbor  latent 
misunderstandings  and  confusions  about  the  nature  of  the 
operational  world.  All  of  this  predicts  to  the  failure 
on  the  part  of  the  researcher  to  incorporate  into  his  or 
her  computer-based  system  the  necessary  user-oriented 
features  so  critical  to  generating  interest,  appeal  and, 
ultimately,  acceptance.  An  anecdotal  example  recalls  the 
first  use  of  the  Joint  Chiefs  of  Staff  (JCS)  regional 
capability  of  the  EWAMS  for  use  by  a  military  analyst, 
who  promptly  asked  why  he  must  relearn  all  the  nations  of 
the  world  in  three  letter  WEIS  representation  rather  than 
the  SOP  military  two-letter  designation. 

1.1. 2. 3  Demonstration  staging.  The  success  or  failure 
of  a  developed  software  product  often  completely  depends 
upon  the  nature  of  the  environment  in  which  the  product  is 
first  introduced  to  a  potential  user.  Nothing  is  so  dis¬ 
connected  from  (or  detrimental  to)  a  presentation  of 
significant  work,  than  to  present  it  in  a  bad  environment. 
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It  most  surely  will  jeopardize  the  acceptance  of  that 
product.  For  example,  imagine  the  effect  of  a  demonstration 
of  EWAMS  in  an  environment  cluttered  by  bearded  professors 
and  disorganized  papers  (a  not  untypical  research  environ¬ 
ment)  versus  one  that  takes  place  in  a  controlled  and 
secure  command-center-like  environment.  The  results  are 
obvious . 


1.1. 3. 3  Premature  demonstrations.  Another  problem 
all  too  often  encountered  is  the  premature  release  and 
demonstration  of  a  product.  Contractors  by  their  nature 
have  very  little  feeling  for  the  proper  timing,  and  in  this 
regard,  often  demonstrate  too  early  with  disastrous  possibly 
even  fatal  results.  A  brief  discussion  of  an  advanced 
computer-based  training  system  failure  will  drive  this  point 
home.  Recently,  during  a  set  of  training  system  demonstra¬ 
tions  to  the  Director  of  DARPA,  one  system  implemented  on 
an  APPLE  II  micro-computer  failed  to  meaningfully  communi¬ 
cate  the  value  of  the  training  aid.  This  resulted  in  an 
unwarranted  skepticism  of  the  value  of  micro-processor-based 
training  aids.  Quite  simply,  this  was  a  problem  which 
could  have  been  averted  by  more  careful  planning  and  evalua¬ 
tion  of  the  system's  "readiness”. 
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1.1.4  Transfer 


1.1. 4.1  Targeting.  Too  often  we  are  not  cognizant  of 
the  overall  requirement  that  we  are  trying  to  address  via 
the  development  of  a  computer-based  system.  That  is  to 
say,  we  must  remain  mindful  of  the  target,  need,  use  and 
problem  to  be  solved,  and  avoid  becomming  overly  impressed 
with  techniques  or  inventive  solutions,  which  although  may 
be  major  breakthroughs  in  and  of  themselves,  must  address 

a  targeted  need. 

1.1. 4. 2  Transfer  site  requirements.  There  exists  a 
significant  problem  in  the  failure  to  adequately  survey 
the  hardware  and  software  at  a  proposed  user's  site,  often 
resulting  in  a  serious  mismatching.  Transfer  efforts 

must  be  done  professionally  and  expertly.  Those  individuals 
responsible  for  the  evaluation  of  the  users  hardware  con¬ 
figuration  and  facilities  have  little  room  for  error.  Be¬ 
ginning  with  the  design  phase,  all  work  should  be  aimed  at 
effective  solutions  to  specific  development  problems.  The 
final  phase  in  the  solution  of  that  problem  must  take  into 
consideration  every  possible  condition  which  could  prevent 
the  integration  of  the  new  work  with  the  existing  technology. 
Problems  encountered  in  the  transfer  process  are  highly 
visible  to  the  ultimate  user  and  may  overshadow  the 
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impression  of  coupe tence  as  well  as  confidence  in  the 
delivered  work.  No  matter  how  valuable  the  analytic  tool, 
if  its  final  installation  appears  to  be  a  comedy  of  errors, 
then  a  value  of  similar  worth  will  be  placed  upon  the  pro¬ 
duct  being  installed.  As  yet  this  situation  has  been  only 
narrowly  avoided. 

1.1. 4. 3  Documentation .  More  commonly,  there  is  a 
failure  to  provide  necessary  instructional  documentation 
along  with  the  software  and  data  sets  being  transferred. 
This  problem  has  the  potential  of  being  as  dangerous  as 
a  faulty  transfer.  Without  the  proper  documentation,  in¬ 
terest  in  the  product  will  wane.  Because  it  is  too  diffi¬ 
cult  to  understand,  it  will  not  be  used.  An  example 
which  fits  the  above  problem  description  occurred  at  the 
very  first  DDF  transfer  to  the  Naval  Postgraduate  School 
(NPS)  in  Monterey,  California.  The  software  and  data 
transferred  was  the  complete  crisis  management  system. 

The  software  products  included  the  URR,  U.S.  Executive 
Aids  (EXECAID) ,  EWAMS,  and  various  utility  programs.  The 
data  sets  transferred  included  the  URR  and  supporting  data, 
EXECAID  and  supporting  data,  and  a  complete  set  of  WEIS 
analytic  and  descriptive  data.  The  installation  of  the 
software  onto  the  NPS  PDP  11  went  reasonably  well.  All 
programs  tested  well  and  transfer  was  complete.  But,  when 
asked  for  more  information  about  how  to  use  all  of  the 
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capabilities  of  the  software — adequate  documentation  was 
unavailable.  When  asked  about  some  of  the  statistical 
techniques  used  by  the  creators  of  the  software,  again 
there  were  problems.  Although  overall  the  transfer  was 
quite  successful,  without  sufficient  documentation  in¬ 
cluding  users  manuals,  sample  output  and  succinct  research 
papers  the  initial  impact  of  the  products  was  minimal. 

1.1. 4. 4  Marketing.  Another  problem  that  can  occur 
is  the  failure  to  adequately  assess  the  bureaucratic 
back-drop  necessary  for  thorough,  formal  transfer.  While 
this  is  not  to  say  that  informal  transfer  is  not  valuable, 
it  is  to  say  that  often  without  top-down  approval  from 
the  "chain-of-command" ,  transfer  could  be  at  best  delayed, 
or  at  worst,  prevented.  A  clear  example  of  this  failure 
to  "market"  a  transfer  at  the  proper  decision-making  levels 
exists  in  the  "continuing"  transfer  of  the  EWAMS  software 
and  data  to  DIA/NMIC.  Just  prior  to  final  transfer  of  the 
software  to  the  NMIC  assurances  were  made  as  to  the  avail¬ 
ability  of  a  "stand-alone"  PDP  11/45  computer  system,  com¬ 
plete  with  assistance  and  guidance  from  the  pentagon  staff. 
This  never  occurred.  The  "stand-alone"  computer  system 
never  became  available,  and  the  DIA  internal  and  contrac¬ 
tor  support  never  materialised.  Even  though  the  DDF  did 
in  effect  manage  to  demonstrate  that  the  software  and  data 
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were  ready  to  be  transferred ,  and  that  it  all  could  be 
done  to  the  original  DIA  specifications,  it  was  still 
delayed  indefinitely.  Why?  Simply  because  of  the 
failure  to  realize  an  everyday  axiom  of  marketing,  that  is, 
to  gain  approvals  from  those  in  key  decision-making 
positions . 


1 . 2  Proposed  Solution 

It  has  become  blear  that  the  research  and  development 
process  especially  as  it  applies  to  the  development  of 
advanced  C  computer-based  systems  has  many  problematic 
areas.  The  cataloguing  of  these  problems  is  the  first 
step  toward  the  framing  of  a  solution.  Step  two  requires 
the  establishment  of  a  set  of  general  and  specific 
mission-oriented  objectives  by  which  the  solutions  can 
be  determined  and  applied.  The  general  solutions  (and 
overarching  project  goals)  appear  below;  some  specific 
solutions  appear  in  section  2.0. 


Objective  1  -  The  establishment  of  a  new  and  vital 
design  phase  for  all  candidate  DARPA/CTO  contractor  soft¬ 
ware  .  A  rigid  set  of  design  standards  will  be  applied  to 
all  new  and  intended  software  products.  Through  the  applies 
tion  of  these  standards  the  software  and  data  requirements 
can  be  categorized  into  groups  which  will  serve  to  indicate 
the  needs  of  the  project  which  are  to  be  provided  by  DDF 
personnel.  Further  analysis  can  be  made  at  this  time  to 
determine  the  pre-estistence  of  data  sets  that  may  fulfill 
the  requirements  of  the  effort.  As  a  function  of  this 
design  phase  clear  cut  alternatives  can  be  examined, 
weighed  and  selected  thereby  insuring  that  the  proposed 


effort  it  both  well  conceived  and  within  the  limits  of 


acceptable  programming  practices. 

Objective  2  -  Provide  a  superstructure  of  effective 
hardware  and  software  development  capabilities.  Every 
effort  will  be  made  to  accommodate  the  hardware  and 
software  requirements  of  the  DDF  user  community.  This 
effort  will  be  centered  around  the  operation  of  DEC  PDP 
11/70  dual  mainframe  computer  systems.  The  full  resources 
of  this  service  will  be  extended  in  an  attempt  to  adequately 
support  the  on-going  development,  design  and  transfer  ef¬ 
forts.  However,  the  provision  of  this  service  is  not  the 
only  goal.  Also  included  in  the  development  phase  are: 
hardware  selection  assistance,  programming  assistance, 
training  and  advice,  creation  of  documentation  standards 
and  a  concerted  effort  to  keep  all  researchers  informed  of 
technical  advances  made  both  inside  and  outside  the  DARP A/CTO 
community. 

Objective  3  -  Take  a  leadership  role  in  the  organiza¬ 
tion  and  presentation  of  professional  and  effective 
demonstrations .  The  advanced  computer-based  systems  and 
data  developed  for  DARPA/CTO  must  endure  intensive  critique 
by  a  viewing  audience.  Therefore,  it  becomes  increasingly 
important  that  a  concerted  effort  be  made  to  establish  and 
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conduct  policies  for  and  training  about  effective 
demonstration* .  Here  too  resides  the  need  for  physical 
as  well  as  consultantive  services.  It  is  very  important 
to  provide  a  well  organized  program  of  demonstrations  in 
an  environment  conducive  to  generating  interest,  appeal 
and  hopefully  acceptance  of  the  research  products  being 
developed  by  those  representatives  of  the  operational 
world  who  may  wish  to  adopt  new  computer-based  systems. 

Objective  4  -  Provide  expertise  ready  to  address 
the  problem  of  transferring  selected  software  and  data 
from  research  status  to  operational  service.  As  part  of 
the  overall  mission  of  DARPA/CTO  successful  research 
products  must,  after  design,  development  and  demonstration, 
be  transferred  to  a  proposed  user's  site.  Care  must  be 
taken  to  adequately  analyze  the  systems  and  facilities 
available  at  the  transfer  site  so  as  to  assure  that  the 
least  amount  of  difficulty  is  encountered  during  the 
transfer.  During  the  transfer  phase  all  supporting  docu¬ 
mentation  should  be  assembled  to  accompany  the  software  and 
data,  comprising  a  complete  package  which  is  useable  and 
understandable.  Also,  care  should  be  taken  to  coordinate 
with  the  transferee  to  insure  that  the  products  being 
delivered  are  what  is  expected  as  well  as  needed. 

Objective  5  -  Create  a  new  management  approach  to  the 
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organization  of  the  DDF.  Most,  if  not  all,  of  the  tech¬ 
nical  problems  currently  facing  the  DDF  user- community 
cannot  be  resolved  by  technical  adjustments  alone.  These 
problems  require  a  new  management  model,  a  re-organization 
to  accomodate  technical  advancement. 

In  the  coming  months--indeed  throughout  Fiscal  Years 
1980  and  1981 — Computer  Systems  Management,  Inc.  will  endeavor 
to  solve  these  problems  through  the  implementation  of  a 
specific  technical/management/administrative  approach.  This 
report  covers  our  first  efforts  and  covers  the  peroid  from 
November  5,  1979  to  December  31,  1979. 


2.0  the  development  of  advanced  c2  computer-based  systems 

2. 1  The  Design,  Development,  Demonstration,  Transfer  and 
and  Documentation  Tasks 

During  the  course  of  Fiscal  Years  1980  and  1981  all  of 
the  tasks  associated  with  computer-based  systems  research 
will  be  addressed.  After  briefly  addressing  the  context  for 
our  work — the  DARPA/CTO  FY80  research  program--we  will  turn, 
in  section  2.1.2,  to  computer-based  systems  development  and, 
in  section  2.1.3,  to  computer-based  systems  demonstration. 

2.1.1  FY80  DARPA/CTO  Research  Program 

It  must  be  noted  that  the  DARPA/CTO  research  program  is 
constantly  evolving.  CSM  is  now  working  from  one  blueprint 
(presented  in  APPENDIX  A  of  this  report) ;  at  the  same  time  we 
continually  monitor  changes  in  the  nature  and  direction  of  the 
program  in  order  to  assess  possible  impact  upon  the  operation 
of  the  DDF. 


2.1.2  Computer-Based  Development 

2. 1.2.1  Hardware ■  As  noted  in  section  1. 1.2.1  the 
selection  of  hardware  is  of  critical  importance  to  the  success 
success  of  a  computer-based  system.  CSM  has  implemented 
a  threefold  strategy  for  the  selection  of  the  most 
appropriate  hardware.  First,  and  as  a  function  of  the 
design  analysis,  CSM  can  recommend  a  specific  hardware 


configuration.  Secondly,  and  with  cost  effectiveness  in 
mind,  CSM  will  canvass  both  the  in-house  and  other  DARPA/CTO- 
owned  hardware  in  an  effort  to  assure  timely  and  appropri¬ 
ate  selection (s) .  Too  often  it  is  the  case  that  hardware 
selection  is  made  solely  on  the  basis  of  what  is  im¬ 
mediately  available  to  the  researcher.  While  this  may 
stimulate  rapid  software  production,  it  often  creates 
sets  of  chain-reaction  problems.  Accordingly,  before 
production  begins,  CSM  attempts  to  match  system  design 
requirements  with  available  (though  not  necessarily  "at 
hand")  hardware.  Finally,  CSM  attempts  periodically  to 
assess  the  "state-of-the-art"  and  the  "state-of-the-art 
likely  to  be",  of  computer  hardware  technology  in  order 
to  avoid  short-sighted  implementation.  In  this  regard, 
periodic  memoranda  are  provided  to  DARPA/CTO  focus¬ 
ing  upon  new  and  innovative  developments  in  computer 
hardware  technology. 

2.1. 2.2  Software .  Software  selection  is  also  a 
function  of  design  requirements,  availability,  and  tech¬ 
nological  advances.  CSM  recommends  to  "developers"  that  com¬ 
puter-based  systems  be  implemented  with  languages  that 
adhere  to  these  three  criteria. 

First,  as  a  function  of  the  design  analysis,  CSM  will 
recommend  a  specific  software  language  compiler  or 


interpreter  to  be  used.  Secondly,  and  with  cost  effective¬ 
ness  in  mind,  CSM  will  evaluate  languages  available  on-line 
versus  those  generally  available  but  not  resident  on  the 
DDF  computer  system  library.  As  in  the  hardware  selection 
process  so  to  it  is  the  case  that  the  software  language 
selection  is  made  solely  on  the  basis  of  what  is  immedi¬ 
ately  available  to  the  researcher.  While  this  may  stimu¬ 
late  rapid  software  production,  it  often  compounds  transfer 
and  conversion  problems  later  on  in  the  life  of  the  soft¬ 
ware  system.  CSM,  therefore,  before  coding  begins, 
attempts  to  match  system  design  requirements  with  available 
(through — again — not  necessarily  "at  hand")  languages. 
Finally,  CSM  attempts  to  periodically  assess  the  state-of- 
the-art  and  the  state-of-the-art  to  be,  of  computer  software 
technology  in  order  to  prevent  short-sighted  implementation. 
Here  too  periodic  memoranda  are  provided  to  DARPA/CTO 
focusing  upon  new  and  innovative  developments  in  computer 
systems  software  languages. 

When  possible,  the  "consistent  strategy"  also  ap¬ 
plies  to  the  selection  of  computer  software.  Too  often, 
systems  have  been  implemented  in  different  languages 
because  they  were  either  available  or  preferred.  Indeed, 
there  are  instances  of  multi-language  system  implementation  . 
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CSM  thus  attempts  to  assure  intra-  and  cross  system  soft¬ 
ware  compatibility. 

CSM  also  attempts  to  develop  and  rigorously  apply 
accepted  standardized  (but  not  inhibiting)  programming 
practices,  such  as  structured,  top-down,  modularized, 
and  flow-oriented  coding  techniques. 

2. 1.2. 3  Coordination.  Through  its  internal  staff, 
a  judicious  set  of  reports  and  memoranda,  and  an  exten¬ 
sive  program  of  seminars,  CSM  attempts  to  coordinate  among 
the  user  and  transfer  community  the  development  of  re¬ 
search  and  applications  software  and  data.  Every  effort 
is  thus  made  to  acquaint  the  DDF  community  with  each 
other's  work.  It  is  hoped  that  through  an  elaborate  set 
of  communications  practices,  the  DARPA/CTO  research  pro¬ 
gram  will  not  be  retarded  by  the  "not  invented  here" 

and  "re-invent  the  wheel"  syndromes.  Clearly,  these  syn¬ 
dromes  are  the  direct  result  of  difficult  "people  problems". 
Through  the  establishment  of  a  genuine  user  camaraderie, 

CSM  has  attempted  to  attack  these  problems. 

2. 1.2. 4  Time-sharing  and  stand-alone  computer  support. 
In  order  to  support  the  design,  development,  demonstation 
(see  section  2.1.3),  and  documentation  (see  section  2.1.4)  of 
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computer-based  systems,  CSM  operates  a  multiple 
DEC  PDP  11/70  timesharing  system  and  other  real-time  or 
stand-alone  micro-processor-based  computer  systems  as 
so  directed  by  DARP A/CTO. 

As  figure  1  suggests  the  computer-based  systems 
development  function  must  be  prepared  to  support  the 
three  possible  paths  of  advanced  research.  First, 

CSM  responds  to  development  requirements  arising  from 
the  typical  user  which  reflect  a  "light"  computer  require¬ 
ment  research  effort.  The  second  path  has  requirements 
which  demonstrate  a  set  of  special  needs,  reflecting  a 
research  effort  relying  more  heavily  upon  the  computer 
than  is  typical.  These  special  needs  may  range  from 
real-time  applications  to  very  large  memory  specifica¬ 
tions,  storage,  or  heavy  central  processing  unit  (CPU) 
utilization.  This  need  can  be  satisfied  by  scheduling 
dedicated  time  on  the  secondary  "heavy  research"  system. 
Third,  a  very  special  applications  requirement  may  arise 
that  cannot  be  easily  categorized  as  either  hetvy  or 
light  computer-based  research,  and  thereby  not  be  assigned 
to  the  primary  or  secondary  system.  The  solution  to  this 
requirement  often  is  found  in  the  selection  of  a  micro¬ 
processor  based  computer  system.  More  and  more  frequently, 
the  hardware  selection  process  defines  a  clear  need  to 
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adopt  the  portability  and  stand-alone  capability  that  a 
micro-processor  can  provide.  CSM  attempts  to  pay  special 
attention  to  the  developmental  needs  of  research  which 
fall  into  the  micro-processor  category. 

The  areas  in  which  CSM  concentrates  primary 
development  support  ensure  that  the  hardware  and  soft¬ 
ware  configurations  are  providing  maximum  utility  to 
the  user  community  are: 

e  Operation  of  the  time-sharing  service  both 
systems  "A  and  B". 

•  Providing  user-community  programming  and 
software  support. 

•  Expanding  the  facility  library  to  track  the 
needs  of  the  users  and  DARPA/CTO. 

•  Providing  engineering  support  to  all  areas 
inside  and  outside  the  DDF. 

•  Providing  new  product  support. 

In  the  area  of  time-sharing  operations  it  is  important 
to  be  aware  that  CSM  operates  a  dual-system  configuration. 
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Systems  "A"  and  "B”  will  be  tied  to  one  another  by  the 
existence  of  a  UNIBUS  peripheral  switch.  The  primary 
function  of  system: "B"  is  to  back-up  system  "A".  Beyond 
that,  system  "B"  is  available  for  "heavy  research". 

CSM  supports  the  operation  of  both  GFE  PDP  11/70* s  as 
one  "complete"  system  even  though  they  may  often  be 
performing  entirely  different  and  separate  tasks.  The 
activities  that  CSM  attempts  to  perform  in  the  operations 
area  include: 

Maintenance  and  operation  of  two  GFE  DEC  PDP  11/70 
mini-computer  systems.  The  operating  hours  are  24 
hours  a  day,  seven  days  a  week  with  one  shift  (prime 
working  hours)  of  attended  operation.  The  remaining 
time  is  unattended.  The  only  exceptions  to  this  schedule 
include  normal  Preventative  Maintenance  (PM)  conducted 
by  the  manufacturer.  Emergency  Maintenance  (EM)  downtime, 
and  at  the  direction  of  the  director  of  DARPA/CTO. 

Performance  of  quarterly  hardware  evaluation. 

This  evaluation  brings  to  the  surface  all  difficulties 
in  "throughput"  as  related  to  hardware  errors,  failures, 
and  configuration  flaws.  Information  from  these  evalua¬ 
tions  is  then  reported  to  the  manufacturer,  CSM  management, 
or  DARPA/CTO  with  recommendations  for  corrective  action. 
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Execution  of  software  programs  to  gather  data  on  the 
usage  and  other  accountable  resources  utilized  by  the  user 
community  on  a  monthly  basis;  and  the  performance  of 
daily,  weekly  and  monthly  system  dumps  to  assure  a  cur¬ 
rent  back-up  of  the  users  files.  The  method  of  back-up 
is  always  efficient  enough  to  insure  that  no  single  user 
could  lose  any  more  than  one-day's  programming  or  data 
collecting  effort,  at  any  time. 

In  the  area  of  software  support  it  also  must  be  noted 
that  more  than  one  configuration  is  being  considered.  The 
primary  system  ("A")  and  secondary  system  ("B")  may  often 
require  redundant  software  capabilities.  However,  there 
is  just  as  great  a  chance  that  system  "B"  will  not  be  run¬ 
ning  UNIX  but  another  operating  system.  CSM  will  support 
both  operating  systems.  Also  CSM  will  stands  ready  to 
support  the  software  needs  of  the  micro-processor  users, 
as  they  are  defined.  Related  to  all  software  activities, 

CSM  attempts  to  take  positive  action  in  the  performance  of: 

•  assistance  to  the  user  community  to  correct 
coding  errors  ("bugs"); 

•  assisting  or  advising  users  in  the  selection 

and  use  of  all  DDF  software  packages  and  languages; 
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•  assistance  to  users  in  the  proper  development  of 
their  applications  software  coding  design  and 
testing ; 

a  responsiveness  in  the  creation  of  new  software 
utilities  by  addressing  the  problems  of  new  or 
modified  software;  and  the 

a  performance  of  ad  hoc  software  performance 

evaluations  to  determine  the  "state-of-the-sof tware" . 
Reports  describing  the  performance  and  existing 
problems  will  be  distributed  in  such  a  manner  as 
to  insure  prompt  attention  to  the  repair  or  re¬ 
placement  of  that  software  routine. 

Because  of  the  ever  heightening  activity  in  computer- 
based  systems  research  the  DDF  facility  library  has  increased 
importance  in  its  role  as  a  repository  of  completed  works. 
Research  conducted  now  will  hopefully  become  the  foundations 
upon  which  future  breakthroughs  will  occur.  Because  of 
this  philosophy,  CSM  is  currently  expanding  enormously 
the  facilities  library  to  include  activities  such  as: 

•  maintenance  of  all  codeable  inventories  and  lists 
of  DARPA/CTO  owned  hardware  and  software  for  in¬ 
stantaneous  retrieval; 
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•  maintenance  of  all  relevant  reports,  sample- 
output,  documentation,  demonstration  aids  and 
codebooks  for  further  distribution  at  the  direc¬ 
tion  of  DARP A/CTO; 

•  creation  of  an  on-line  system  tutorial  to  assist 
in  the  education  of  new  users,  as  to  the  policies, 
procedures,  and  capabilities  of  the  CSM/DDF.  Also 
included  should  be  sound,  informative  sections  on 
packages,  languages  and  utilties  resident  on  the 
primary  system;  and  the 

•  maintenance  of  data  and  program  inventories  and 
biographies  for  shared  access  by  the  user  community. 

Especially  because  of  the  size  and  complexity  of  the 
CSM/DDF  operation,  it  is  becomming  increasingly  more  appro¬ 
priate  to  become  less  reliant  upon  the  services  of  outside 
or  even  non-DDF  personnel.  Therefore,  CSM  believes  that 
an  engineering  support  function  should  be  included  in  the 
development  function.  The  very  first  effort  in  this  re¬ 
sponse  area  is  the  construction  of  a  new  and  environmentally 
sound  computer  room.  Since  nearly  all  of  the  existing  DDF 
inventory  is  government  furnished  equipment  (GFE)  it  is 
essential  that  it  be  given  the  undivided  attention  of 
specially  qualified  personnel.  Other  than  the  initial  phase 
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of  computer-room  construction,  CSM  believes  the  following 

activities  should  be  conducted  under  engineering  sunport: 

e  Electronic  component  parts  repair; 

e  Facility  layouts,  and  design  in  the  areas  of 
drawings,  construction,  mechanical  and  elec¬ 
trical  plans; 

e  Terminal  repairs  and  renovations  that  require 
few  spare  parts  inventories; 

e  Power  consumption  level  monitoring  and  conser¬ 
vation; 

e  Special  wiring,  cabling,  and  inter  device/system 
connections ; 

e  Special  small  projects  design,  (e.g.,  small 
relays  for  voice  actuated  system,  or  control 
podium  for  demo  room) ;  and 

e  Communications  equipment  support  and  performance 
of  quarterly  status  evaluations.  These  reports 


are  to  be  forwarded  to  both  CSM  and  DARPA/CTO 
management  explaining  difficulties  and  alterna¬ 
tive  solutions. 

Finally,  new  products  are  always  appearing  in  the 
market  place.  It  is  the  function  of  computer-based  systems 
development  to  be  aware  and  educated  about  the  capabilities, 
weaknesses,  cost  and  availability  of  these  new  devices. 

CSM  believes  that  the  PFC  PDP  11/70  may  very  well  be  the 
"dinosaur  ot  the  1980's".  We  feel  that  it  is  very  impor¬ 
tant  to  be  prepared  for  an  11/70  replacement.  By  doing 
this  we  will  facilitate  a  smooth  "down-grade"  toward  what 
the  technology  future  has  already  predicted.  More  speci¬ 
fically,  CSM  will  as  a  function  of  the  role  which  it  pro¬ 
poses  to  play  for  DARrA/CTO  through  the  DDF,  examine  in  t imc 
the  full  range  of  implications  for  the  development  of 
computer-based  systems  which  have  already  been  suggested 
by  the  current  revolution  in  very  large  scale  integration 
(VLSI) . 

2.1.  *  Demons  t ration 

If  one  is  at  all  familiar  with  the  DDF,  one  realises 
the  importance  of  the  role  of  the  demonstration  at  the  DDF. 

A  new  and  exciting  era  is  about  to  dawn  in  computer 


audio-visual  techniques  via  the  first  production  version 
of  the  special  data  base  management  system  (SDMS) .  More 
importantly,  the  SDMS  is  coming  to  the  DDF.  This  will 
open  up  entire  new  vistas  of  demonstrational  techniques 
and  the  realization  of  a  demonstration  that  just  cannot 
be  ignored.  This  exciting  event  will  further  propel 
the  DARPA/CTO  into  the  services  and  acceptance 
of  its  products  as  "real"  research.  Moreover  the  need 
for  a  particularly  professional  demonstration  room 
becomes  apparent.  CSM  intends  to  accept  as  GFE  delivery 
the  Computer  Corporation  of  America's,  Inc.  version  of 
the  SDMS  in  its  entirety  (both  hardware  configuration 
and  software) .  This  will  require  the  creation  of  addi¬ 
tional  environmental  facilities  in  the  new  CSM/DDF  computer 
room  and  demo  room  to  accommodate  the  DEC  PDP  11/70  used 
by  the  SDMS.  CSM  has  already  procured  power, 
environment  and  space  to  demonstrate  the  system.  CSM 
is  also  prepared  to  assist  in  the  transfer  of  all  cur¬ 
rently  demons tratable  software  and  data  from  the  primary 
time-sharing  system  to  the  SDMS. 

2. 1.3.1  Opposing  cultures.  In  order  to  assure  that 
applied  computer-based  systems  are  tailored  during  the 
design  and  development  stages,  CSM  intercedes  (only) 
at  the  direction  of  DARPA/CTO  during  these  stages  to 
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evaluate  systems  development  against  perceived  user  needs 
and  requirements.  At  a  general  level  this  will  result 
in  the  application  of  known  and  standardized  user-oriented 
programming  techniques ,  including  ease  of  input,  menu 
selection,  and  effective  display,  among  other  features. 

At  the  more  specific  level,  attention  is  devoted  to 
the  particular  intended  use  of  the  system  to  be  demon¬ 
strated.  For  example,  demonstrations  of  systems  designed 
to  enhance  I&W  capabilities  will  be  conducted  around  an 
environment  and  senarios  familiar  to  the  intended  user. 

At  the  direction  of  DARPA/CTO,  CSM/DDF  staff  demonstrates 
extant  "products”  to  prospective  transferees.  Such  demon¬ 
strations  must  thus  be  conducted  only  by  those  cognizant 
of  the  users  requirements  and  then  only  after  a  background 
analysis  has  been  conducted  of  the  users  environment.  It 
is  hoped  that  such  rigorous  analytical  preparation  will 
minimize  user  indifference  and  hostility.  (While  we  do 
not  presume  "routine"  user  hostility,  we  must  nonetheless 
assume  that  the  introduction  of  advanced  technology  to 
highly  proceduralized  environments  is  delicate.) 

2. 1.3. 2  Demonstration  staging.  Extending  from  our 
concern  about  the  inherently  opposing  cultures  is  a  con¬ 
cern  for  the  physical  environment  in  which  the  demonstra¬ 
tion  takes  place.  Accordingly,  very  special  care  must  always 
be  taken  to  create  a  positive  professional  demonstration 
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milieu.  At  the  most  basic  level,  this  care  is  mani¬ 
fest  in  an  overt  marketing  strategy,  from  which  a  user  is 
"handled"  in  a  manner  appropriate  to  his  rank  and  respon¬ 
sibility. 


2. 1.3. 3  Premature  demonstrations.  In  an  effort  to 
avoid  at  all  cost  a  demonstration  failure  resulting  from 
the  premature  release  of  a  computer-based  system,  CSM 
has  developed  and  applied  a  set  of  demonstration 

"readiness  criteria".  Specifically,  the  criteria  are 
relevant  to  the  "state-of-the-system"  in  the  following 
ways:  "Has  the  system  been  thoroughly  checked  for  all 

software  errors  and  bugs?"  "Has  a  coherent  demonstration 
sequence  been  pre-tested?"  "Can  the  user  operate  the 
system  easily  by  himself? *  "Is  the  system  flexible  enough 
(and,  where  appropriate,  are  the  data  sets  extensive 
enough)  to  permit  a  flexible  demonstration?"  "Have  back¬ 
up  technical  and  non-technical  materials  been  adequately 
prepared?"  "Have  likely  questions  been  anticipated?" 

"Has  the  back-up  system  been  thoroughly  tested?"  "Has 
supporting  (carry-away)  documentation  (sample-output  and 
user's  manuals)  been  prepared?"  (See  APPENDIX  C) 

2.1.4  Documentation. 


If  all  of  this  is  to  be  routinized,  we  must  begin  to 
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document  all  of  the  problems  and  progress  made  toward  the 
development  and  transfer  of  useful  C  computer-based  aids 
of  all  kinds.  CSM  has  prepared  and  submitted  a  plan  for 
such  documentation.  It  appears  in  APPENDIX  B. 
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3.0  CONCLUSION 


This  first  Quarterly  Technical  Report  has  examined  the 
issues  of  computer-based  development,  demonstration,  and  docu¬ 
mentation  connected  with  the  design,  development,  demonstration, 

2 

and  transfer  of  advanced  command  and  control  (C  )  computer- 
based  information,  decision,  forecasting,  training  and  readi¬ 
ness  systems.  Future  reports  will  deal  with  design  and  transfer 
and  other  related  issues. 
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4.0  FOOTNOTES 


*  The  TRAP  system  was  developed  by  CACI,  Inc.  for  DARPA/CTO. 
2 

Similar  problems  occurred  in  connection  with  the  develop¬ 
ment  of  the  DARPA/CTO  Early  Warning  and  Monitoring  System 
(EWAMS)  and  the  Executive  Aids  for  Crisis  Management. 

3 

See  S.  J.  Andriole,  Progress  Report  on  the  Development  of 
an  Integrated  Crisis  Early  Warning  System,  Decisions  and 
Designs,  Inc.,  McLean,  Virginia,  December,  1976;  and 
Judith  Ayres  Daly  and  Thomas  R.  Davies,  The  Early  Warning 
and  Monitoring  System:  A  Progress  Report,  Decisions  and 
Designs,  Inc.,  McLean,  Virginia,  July,  1978. 

*  UNIX  is  a  trademark  of  the  Western  Electric  Company.  It 
was  developed  at  Bell  Laboratories  by  Kenneth  Thompson 
and  Dennis  Ritchie. 

5  See  Gerald  W.  Hopple,  Final  Report  of  the  Cross-National 
Crisis  Indicators  Project,  University  of  Maryland,  College 
Park,  Maryland,  forthcoming. 
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The  ultra-rapid  reader  (URR)  was  developed  by 
Perceptronics,  Inc.  for  DARPA/CTO.  Succinctly,  it  is 
a  system  for  rapid  reading  which  presents  text  in  short 
bursts,  one  word  at  a  time  in  the  center  of  a  CRT. 

The  technique  enables  a  user  to  focus  his  or  her  eyes 
in  one  position  and  not  move  them  as  the  words  appear 
one  at  a  time  on  the  screen.  See  Steven  Levin,  The 
Ultra-Rapid  Reader,  Perceptronics,  Inc.,  Woodland  Hills, 
California,  February,  1979. 

^  The  WEIS  data  set  contains  approximately  24M  bytes. 

8  The  software  problem  is  so  visible  in  the  DoD  that  it 
attracks  a  disproportionate  amount  of  attention  (in  terms 
of  dollar  investment)  each  year  during  Congressional 
budget  testimonies. 

9  Bolt,  Berenek,  and  Neuman,  Inc.  (BBN) ;  National  Security 
Agency  (NSA) ;  Information  Science  Center  (ISC). 

l0Prudence  dictates  that  version  7  of  UNIX  be  scrutinized 
carefully  before  implementation  in  order  to  avoid  un¬ 
necessary  problems. 
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DoDl  is  the  intended  (higher  order)  language  to  be 
adopted  by  DoD  as  DoD  standard  at  some  point  in  the 


future. 

12  see  James  J.  Allen,  MCCRESSA  Users  Guide,  Computer 
Corporation  of  America,  Arlington,  Virginia,  August, 
1978. 

13  see  Leo  Hazlewood,  et.al.,  Planning  for  Problems  in 
Crisis  Management,  CACI ,  Inc.,  Arlington,  Virginia, 
June,  1977  and  Barry  M.  Blechman  and  Stephen  S.  Kaplan, 
Force  Without  War,  The  Brookings  Institution,  1978. 

See  Gary  M.  Guilbert,  The  GEN  System  for  Entering, 
Validating,  Updating  and  Reporting  of  Events  Data, 
International  Relations  Research  Institute,  University 
of  Southern  California,  August,  1978. 

1^  See  Ward  Edwards,  How  to  Use  Multi-Attribute  Utility 
Measurement  for  Social  Decision-Making,  Social  Science 
Research  Institute,  University  of  Southern  California, 
August,  1976  and  Dennis  M.  Buede  and  Janice  E.  Ragland, 
Cost-Benefit  Analysis  Applied  to  the  Program  Objectives 
Memorandum  (POM) ,  Decisions  and  Designs,  Inc.,  McLean, 
Virginia,  November,  1978. 
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RESEARCH  PROGRAM 


45 


THE  FY80  DARPA/CTO  PROGRAM 


The  Computer  Systems  Management,  Inc.  (CSM)  technical 
approach  is  a  function  of:  (1)  the  structure  and  direc¬ 
tion  of  the  overall  CTO  research  program,  and  (2)  the 
computer-based  systems  design,  development,  demonstration 
and  transfer  requirements  which  flow  from  that  program. 

The  DARPA/CTO  Research  Program 


Figure  presents  the  current  CTO  organization  by 
function  and  program  thrusts.  As  stated  in  section  1.0 
above,  the  DARPA/CTO  has  as  its  mission  the  development, 
application  and  transfer  of  computer-based  systems  for 
improved  DoD  information  management  and  display,  forecast¬ 
ing,  decision  making,  training,  and  human  performance 
especially  as  such  activities  occur  in  command  and  control 
environments.  More  specifically,  this  mission  is  realized 
through  distinct  research  programs. 

2 

•  Command  and  Control  Information  Systems  (C  IS) 

2 

-  C  IS  consists  of  four  main  research  thrusts: 
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spacial  data  base  management,  advanced  map  displays, 
teleconferencing  and  man-machine  relations.  Each  of 
these  areas  of  research  aims  at  the  development  of 
basic  technology  for  new  and  unique  man-machine  systems 
to  aid  command  and  control  and  intelligence  (Cl) . 

Spacial  data  base  management  is  a  new  technique  for  the 
retrieval  of  information  from  computerized  data  bases. 
Information  is  retrieved  by  its  location  and  appearance, 
relying  on  human  spacial  memory  functions.  The  advanced 
may  display  research  is  developing  technology  for  the 
creation  of  realistic  computer-generated  images,  movies 
and  video-disk-based  photographic  systems  for  presenting 
geographic  information.  Teleconferencing  research  is 
developing  a  system  for  high  fidelity  distributed  con¬ 
ferencing.  Research  in  the  area  of  man-machine  relations 
focuses  upon  the  development  of  basic  technology  for  im¬ 
proving  DoD  "attitudes"  toward  computer  use. 

2 

Some  of  the  systems  developed  thus  far  in  the  C  IS 
program  include  a  prototype  spatial  data  base  management 
system,  a  PDP  11-based  "production"  SDMS,  an  adaptive 
(intelligence)  information  filter,  an  automated  desk  SDMS, 
a  computer  based  message  typography  system,  a  group 
decision  aid,  computer  generated  (from  digital  data  bases) 
movie  maps,  interactive  video-disk-based  photographic 


movie  maps,  and  an  ultra-rapid  reader.  Many  of  these 
systems  reside  at  the  DDF;  the  remainder  are  intended 
as  additions  to  the  DDF  demonstration  program. 

e  Command  and  Control  Decision  and  Forecasting 
Systems  (C2D4FS) . 

The  C  DiFS  program  attempts  to  develop  new, 
and  improve  old,  methodology  for:  (1)  ISW  and  (2)  opera¬ 
tions.  The  objective  in  the  I4W  thrust  of  the  program  is 
to  develop  and  improve  methodologies  for  forecasting, 
estimate  generation,  and  assessment  of  soft,  "non-quanti- 
fiable"  data.  On  the  operational  side,  research  is  intended 
to  develop  and  test  technologies  and  methods  for  improved 
decision-making  and  rapid  option/action  selection.  These 
two  new  over- arching  initiatives,  when  combined  with  the 
integration  of  basic  research  to  improve  decision  and 
forecasting  systems  in  general,  seek  to  enhance  I&W  and 
operations  functions  and  thereby  improve  command  and  con¬ 
trol  on  all  levels. 

Some  of  the  computer-based  systems  developed  thus 
far  include  the  EWAMS,  three  executive  decision  aids  for 
crisis  managers  relevant  to  U.S.  actions/objectives,  prob¬ 
lems,  and  descriptions  of  past  crises,  a  Soviet  crisis 
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management  decision  aid,  a  data  entry  utility  program,  a 
counter-terrorism  research  aid,  and  numerous  other  analytical 
programs.  The  C^d&^S  program  has  also  produced  numerous 
empirical  data  bases.  Nearly  all  of  these  (software  and 
data)  systems  now  run  on  the  DDF;  others  are  awaiting 

«> 

implementation,  while  still  others  are  in  the  planning 
phase. 

The  contribution  of  the  former  Advanced  Decision 
Technology  (ADT)  program  deserves  special  mention.  Selected 

* 

computer-based  systems  include:  OPINT,  EVAL,  POM  &  SCORE. 
OPINT  is  an  aid  to  decision  making  when  the  key  variables 
are  unknown.  OPINT  provides  dyadic  influence  diagramming 
to  aid  decision  makers  in  the  selection  from  various  options. 
EVAL,  does  hierarchical  decomposition  evaluation  of  com¬ 
plex  systems.  The  user  can  create  models  and  assign 
weights  to  various  components  thereby  assigning  its  im¬ 
portance.  The  system  generates  a  final  score  and  produces 
the  ability  to  detect  the  key  factors  which  were  significant 
in  the  ultimate  score.  POM,  (Program  Objectives  Memorandum) 
assists  in  budgeting  analysis  by  generating  a  profile  of 
(.  items  considering  both  cost  and  effectiveness,  and  the 

rationale  used  in  the  determination  of  this  effectiveness. 

POM  is  generally  most  useful  in  budgeting  cycle  areas, 
t  SCORE,  is  an  application  which  implements  a  computer-based 

scoring  rule  technique.  A  trainee's  ability  to  give 
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accurate  probabilistic  assessments  to  a  series  of  mul¬ 
tiple  choice  questions  is  tested.  After  several  tests, 
the  trainee  has  usually  learned  to  give  significantly 
better  assessments.  All  of  this  Advanced  Decision 
Technology  software  will  reside  on  the  DDF  in  FY80. 

e  Advanced  Training  and  Human  Performance 
Technology  (AT&HPT) 

-  AT&HPT  has  a  dual  mission:  (1)  develop 
advanced  training  methodologies  and  computer-based 
systems  for  the  maintenance  and  operation  of  complex 
military  systems  and,  (2)  to  understand  both  the  limits 
of  human  performance  and  methods  for  extending  them.  At 
a  time  when  human  performance  is  treated  as  an  after¬ 
thought  in  the  design,  maintenance,  and  operation  of 
military  systems  it  becomes  increasingly  more  evident 
that  an  increasing  need  for  the  upgrade  of  the  quality 
of  manpower  capabilities  is  paramount.  Initial  research 
has  shown  promising  returns  in  training,  selection  and  job 
design  methodology  as  investments  are  made  to  better  under 
stand  the  limits  of  human  performance  capabilities. 

Most  of  the  computer-based  systems  developed  in 
AT&HPT  may  be  viewed  as  extensions  from  the  PLATO  IV  and  V 


computer-assisted  instruction  (CAI)  systems.  However,  the 
current  emphasis  is  upon  the  development  of  low-cost,  rugged,  and 
distributed  micro-processor  based  systems. 

All  of  these  research  programs  thus  call  for  the  devel¬ 
opment  of  advanced  computer-based  systems  across  a  wide  number 
of  areas.  Computer  Systems  Management,  Inc.  (CSM)  intends 
to  support  for  twenty- four  months  the  design,  development, 
demonstration  and  transfer  of  these  systems  for  DARPA/CTO. 
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APPENDIX  B 

A  PLAN  FOR  THE  DOCUMENTATION  OF 
DARP A/CTO- SUPPORTED  RESEARCH 
AT  THE  DARP A/CTO/DDF 


A  PLAN  FOR  THE  DOCUMENTATION  OF  DARPA/ 
CTO-SUPPORTED  RESEARCH  AT  THE  DARP A/CTO /DDF 


James  F.  Wittmeyer,  III 


(am. 

Computer  Systems  Management,  Inc. 

1300  WILSON  BOULEVARD.  SUITE  103 
ARLINGTON,  VIRGINIA  22100  •  (7031 0204MI 


1.0  INTRODUCTION 


Over  the  years  DARPA/CTO  has  supported  numerous  research  proj¬ 
ects  in  the  C?  information,  decision,  forecasting,  training,  and 
combat  readiness/effectiveness  areas.  Much  of  this  support  has  been 
devoted  to  the  development,  testing,  and  transfer  (to  the  Services 
and  the  Intelligence  Community)  of  computer-based  systems  and  com¬ 
puter  aids  while  other  support  has  been  devoted  to  more  basic  re¬ 
search  necessary  for  the  development  of  such  systems  and  aids. 

Since  much  of  this  research  is  cumulative  in  nature,  and  since  it 
is  conducted  in  divergent  fields  and  disciplines,  it  is  necessary 
to  establish  procedures  for  acquiring,  storing,  documenting,  and 
distributing  the  information,  reports,  software,  and  hardware  con¬ 
nected  with  the  research.  This  plan  thus  presents  a  number  of  ideas 
and  suggestions  for  gathering,  processing,  and  distributing  infor¬ 
mation  important  to  the  chronology  and  progress  of  DARPA/CTO,  ideas 
and  suggestions  which  will  be  implemented,  modified,  or  dismissed 
at  the  direction  of  DARPA/CTO. 


2 . 0  APPROACH 


2.1  Problem  Scope 

DARPA/CTO  generated  information  takes  many  forms,  including 

e  Written  Technical  Reports; 
e  Written  Technical  Memoranda; 
e  Filmed  Technical  Reports; 
e  Filmed  Technical  Memoranda; 

e  Written  Demonstration  Scenarios  and  "Sample" 

Output ; 

e  Filmed  Demonstrations; 
e  Minicomputer  Software; 
e  Microcomputer  Software; 

e  Seminar/Workshop/Conference  Proceedings; 
e  Minicomputer  Hardware  Configurations; 
e  Microcomputer  Hardware  Configurations; 
e  Other  Hardware; 

•  Miscellaneous  Briefing  Materials;  and 

•  Funded  Technical  Proposals. 


All  of  this  information  could  easily  overtake  one's  organiza¬ 


tional  resources.  Since  DARPA/CTO  is  not  structured  to  maintain 
such  information  beyond  its  own  purposes,  CSM  here  proposes  to  as¬ 
sist  DARPA/CTO  with  this  information  management  problem. 

2.2  Proposed  Solution 

2.2.1  Information  Organization 

With  DARPA/CTO  approval,  CSM  proposes  to  organize  the  infor¬ 
mation  according  to  the  current  CTO  program  thrusts  cross-referenced 
by  CTO  contractor.  This  will  facilitate  the  organization,  retrieval, 
and  distribution  of  information  in  a  timely  and  efficient  manner. 

In  addition,  fixed  categories  will  be  set  up  in  an  effort  to  main¬ 
tain  consistent  documentation  procedures,  as  suggested  below. 
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Program:  C2  Information  Systems 


It  is  recommended  that  the  matrices  for  all  of  the  CTO  pro¬ 
grams/information/contractors  be  computerized  into  a  master  in¬ 


ventory  which  would  guide  the  collection  and  maintenance,  and  dis¬ 
tribution  of  materials.  INGRES  could  easily  be  used  to  cross-ref¬ 
erence  the  inventories  for  information  type,  contractor,  and/or 
program  or,  just  as  easily,  a  small  utility  program  could  be  written. 

2.2.2  Information  Collection 


There  is  no  question  that  CSM  will  need  the  assistance  of 
DARPA/CTO  to  gather  the  suggested  information.  Fortunately,  the 
CTO/CSM/DDF  already  has  a  good  number  of  reports  and  other  docu¬ 
ments  gathered  over  the  past  three  to  four  months.  These  reports 
have  been  cataloged  and  organized  for  easy  library-like  retrieval. 
They  have  not  been  computerized,  however.  Also,  the  annual  CTO 
Hardware/Software  Inventory  has  yielded  a  good  deal  of  information 
about  hardware  and  software  configurations.  Finally,  the  DDF  equip¬ 
ment  inventory  constitutes  another  excellent  source  of  information. 

In  order  to  gather  and  organize  information  beyond  DDF's  im¬ 
mediate  resources,  the  CTO  contractor  community  will  have  to  be 
contacted  and  be  willing  to  cooperate.  In  the  past,  contractor 
response  has  been  at  the  60%  level.  This  response  rate  will  have 
to  increase  if  we  are  to  assemble  a  representative  collection  of 
materials;  certainly  "encouragement"  from  CTO  will  greatly  improve 
our  chances  for  success. 
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Finally,  for  organization  purposes  we  feel  that  a  test  case 
involving  either  a  single  contractor  (like  MIT  or  Perceptronics) 
or  one  CTO  program  should  be  attempted  first  so  we  can  identify 
the  issues  and  problems  relevant  to  the  collection  and  organiza¬ 
tion  process. 


3 . 0  TIMETABLE 


The  target  date  for  the  computer-based  working  information 
management  system  has  realistically  been  set  for  March  31,  1980. 
However,  in  order  to  meet  this  date  we  will  require  prompt  response 
from  the  contractors  and  our  consultants.  If  we  are  able  to  re¬ 
ceive  the  information  from  the  contractors  (including  lists,  reports, 
films,  and  the  like)  within  sixty  (60)  days,  then  we  can  maintain 
the  schedule.  If  not,  then  our  schedule  will  slide  accordingly. 
Further,  with  good  cooperation,  we  should  be  able  to  get  a  single 
contractor  or  program  system  running  by  January  31,  1980. 


4.0  CONCLUSION 


All  of  this  is  subject  to  DARPA/CTO  review.  We  are  anxious 
to  build  the  proposed  system  and  invite  comments,  criticisms,  and 
suggestions. 


APPENDIX  C 

READINESS  STATUS  REPORTING  FOR  THE 
DEMONSTRATION  OF  ADVANCED  C2  COMPUTER-BASED  SYSTEMS 


i. 

! 

. 

MONTHLY  SOFTWARE  DEMONSTRATION  AND  TRANSFER 

' 

* 

SUMMARY 

f 

Data:  February,  1980 

Transferable 

t 

Software  Systems: 

Demons tratable 

Early  Warning  System 

Yes 

No* 

Executive  Aids 

Yes 

No* 

1 

TRAP 

NO* 

1 

No* 

OPINT 

No* 

No* 

EVAL 

No* 

No* 

RAM 

No* 

No* 

SCORE 

No* 

No* 

DECISION 

No* 

No* 

Group  Decision  Aid 

No* 

No* 

STEAMER 

No* 

No* 

PLATO 

No* 

No* 

** 

PRESS 

No* 

No* 

AIS 

No* 

• 

No* 

Computer  Generated  Maps 

No* 

No* 

Duncan/Job  Estimator  No*  No* 


*  Saa  attachad  elaboration 


MONTHLY  SOFTWARE  DEMONSTRATION  AND  TRANSFER  READINESS  REPORT 


Date:  February,  1980 

System 

Early  Warning  System 

Executive  Aids 

TRAP 

OPINT,  EVAL,  RAM, 
SCORE,  DECISION 

Group  Decision  Aid 

STEAMER 

PLATO 

PRESS 


ELABORATION 


Status 


Demonstratable  primarily  because  of 
continued  familiarity  with  system; 
no  back  up  slides;  no  draft  system 
specification  or  functional  descrip¬ 
tion;  no  internal  documentation. 


No  draft  system  specification, 
functional  description,  or  internal 
documentation;  no  back-up  demonstra¬ 
tion  slides;  no  continued  training 
on  capabilities  or  evolution  of 
system (s) . 


Unclassified  version  not  supported  by 
training,  viewgraphs,  reports, 
internal  documention,  system 
specification,  or  functional 
description. 


No  software  delivered;  no  other 
supporting  documention  except  DDI's 
documention  which  is  generic  and 
(probably)  peripheral  to  software. 


No  software  documention  of  any 
kind;  no  reports;  no  viewgraphs. 


No  documention  of  any  kind. 


No  documention  of  any  kind. 


Minimum  software  documention;  no 
slides;  no  reports;  cursory  user's 
"guide". 


Status 


Computer-Generated  Maps 
Duncan/Job  Estimator 
Ultra-Rapid  Reader 


Undelivered. 

No  documention  of  any  kind. 

No  documention  of  any  kind. 

No  documention  of  any  kind  except 
for  a  cursory  user's  manual. 


**Special  Note:  Obviously  DDF  personnel  can — and  have — given 
demonstrations  of  some  of  the  above  listed  undemonstratable 
software  systems  without  the  requisite  back-up  materials. 
However,  as  we  hope  is  clear,  CSM  will  not  always  be  able  to 
give  proper  demonstrations  without  the  necessary  back-up 
and  training.  Transfer  is  much  more  difficult  without 
good  documention.  We  believe  that  totally  effective  demonstra¬ 
tions  can  only  be  given  when  all  materials  and  serious 
training  has  been  delivered. 


